1. Field of the Invention
The present invention relates to the gathering and analysis of market transaction data, where such transactions are contemplated or completed by electronic means, and specifically to the use of software agents to represent and to assist the activities of consumers and providers within an electronic “virtual marketplace”.
2. Description of Related Art
The trading of goods and services is one of the basic human activities. From the first meeting of pre-historic men to trade animal skins for berries, to country fairs, to the most intricate computer-based trading of international financial instruments, the marketplace supports the survival and flourishing of our lives.
Every era has established a marketplace based on the technology of its time. Early marketplaces were locations for face-to-face meetings between people offering items for trade. The development of printed material and postal correspondence made it possible for people to trade without ever meeting in person. In modern times, a sophisticated division of labor system involving producers, suppliers, distributors, advertisers, market researchers, payment clearinghouses, customers, and many others, supports our economic lives.
The flow of goods and services, however, is not the whole story. The people and organizations on the production, distribution, sales, and marketing side 1 of trade (“providers”) need to understand what goods and services are desired by buyers. The people and organizations who purchase or acquire offered items (“consumers”) need to learn what goods and services are available. The flow of this market information is critical to the successful operation of a market.
A brief consideration of modern methods of commerce shows us that consumers have many ways to gather information about available goods and services. Some of these are: visiting stores; browsing catalogs; viewing advertisements on television, on billboards, and in magazines and newspapers; soliciting recommendations from friends; and receiving unsolicited mail advertisements.
We also observe that vendors and other providers use many avenues to disseminate information about available goods and services. These include: broadcast and direct advertisements; in-store displays; telephone solicitations; and so on.
Providers also try to collect information about consumers' desires and buying habits so that they can better serve their current customers and gain new ones, thereby enhancing profits. Current market research methods include: keeping current customer purchase data on file; buying or renting mailing lists of other vendors; listening to focus groups; running pilot sales in test markets, and so on.
However, even with so many pathways for information exchange, there are many disadvantages inherent in the current methods of commerce.
Disadvantages of Current Methods of Commerce
Although consumers clearly reap many benefits in today's marketplace, there are still many disadvantages that lead to consumer frustration. Here are some obvious disadvantages for consumers:                Just collecting basic information about available goods and services, their features, and their prices is tedious and time-consuming. To find a particular product or verify availability, price, and features, the consumer typically may need to visit several stores across a wide area. Another consumer may choose to spend hours on the telephone, perhaps incurring various communication charges, navigating irritating automated attendants, trying to reach a human who can answer questions. Yet another consumer will peruse stacks of catalogs and accumulated direct mail advertisements. This process wastes the consumer's valuable time, especially when the search falls.        Information isn't always available when it's needed. Consumers frequently rely on ephemeral television and radio advertisements to learn of products and special promotions. But these ads seldom arrive when the consumer is ready to make a selection. Even with print ads, the information is soon lost as the stack of newspapers is carried away for recycling.        It is very difficult for consumers to pick out items of interest from the daily bombardment of advertising. Television and radio spots, billboards, street corner kiosks, the daily newspaper, direct mail coupon packages, in-store merchandise displays, magazine advertisements and inserts, posters on and in mass transportation vehicles, storefront banners—these advertisements and a host of others vie for the consumer's attention daily. Marketing research firms use the term “impression” to denote one occurrence of a person perceiving an advertisement. Some authorities claim that an urban consumer in New York City receives up to 15,000 impressions per day! It is no wonder that consumers become desensitized to the advertising around them, and fail to see the items that are truly of interest.        Consumers have little or no control over the timing or presentation of advertising. They can't arrange to receive information at a convenient time. Telephone solicitors always seem to interrupt dinner. The radio advertisement for an anticipated concert is on the air during “drive-time”—when the consumer is driving and can't write down the phone number of the ticket agency. Consumers can't even choose a preferred format for receiving advertisements. If the merchant advertises only in broadcast media, the consumer won't be able to find the information in today's newspaper. Sorting through the barrage of direct mail may require more time than the consumer can allocate to this task, and so the consumer discards a potentially useful notice.        A consumer usually receives no direct benefit for inspecting a provider's advertisements, except for the information itself if the advertisement is relevant. Occasionally, providers and marketing research firms give consumers a small fee or gift in return for participating in a survey. This is called “paying a consideration.” However, there are few opportunities for consumers to receive considerations, and no way for consumers to seek out providers that are willing to pay considerations.        When a consumer has an immediate need for product or service information, it may be nearly impossible to gather the information quickly. This is especially true when the consumer doesn't know where to look for the information, or doesn't have quick and convenient access to sources of information.        Many consumers rely on the recommendations or evaluations of third parties to help them make buying decisions. Consumer rating guides and endorsers provide this kind of information, but the information is not always available at the time or place of the consumer's purchase decision.        
Providers, too, have their share of frustrating difficulties using the current methods of commerce. Some of the disadvantages for providers are:                Much of the demographic data from traditional sources is out-of-date or incorrect. Providers frequently buy or rent magazine subscription lists and other providers' customer lists. However, a name on a list is not necessarily removed when the individual no longer subscribes or is no longer a customer. It is expensive to examine lists for data entry errors, duplicates, minor spelling variations, and so on. When a provider exchanges a list with another provider, who exchanges with a third provider, who exchanges with yet another provider, a provider can easily end up with a list that contains a high proportion of names of consumers who are not at all interested in the provider's goods.        It is expensive to target advertisements to specific customers. As mentioned above, mailing lists frequently contain a high percentage of names of uninterested customers, yet buying or renting mailing lists is costly. Controlled circulation magazines, where potential subscribers complete a qualification form to receive a (usually complimentary) subscription, don't always yield truly qualified names, since there typically is no mechanism to verify the information that the potential subscribers report.        It is difficult for a provider to deliver information directly to a consumer who is ready to buy. Even if providers could easily identify those consumers, there is a time lag for the delivery of pertinent information. For example, direct mail requires planning weeks in advance. An in-store salesperson can assist the consumer, but only if the consumer has previously learned that the store carries suitable products. Providers need a mechanism for delivering information to consumers precisely at the moment when it is most helpful.        It is difficult and costly to personalize information to a consumer. Most consumers are unimpressed by feeble attempts at personalization such as form letters that read “Dear MR. JONES JOHN, The JOHN FAMILY may have already won . . . ”        It is especially difficult to determine what specific interests a particular consumer has. Most data of this nature must be inferred from subscription lists, member lists, spotty purchase history, etc. It is rare that a consumer directly informs a provider of a particular interest.        Providers have little control over the timing of the delivery of their printed advertising messages. The use of special mailing classes for bulk mail to reduce mailing costs results in erratic delivery times. In the United States, providers using Third Class mail cannot pinpoint even the week that the mail will be delivered. For example, sometimes consumers don't receive a special sale notice until after the sale date. Also, mailed notices must be prepared well in advance to take advantage of bulk mail, so the provider's quick response to market conditions is impeded.        It is difficult for providers to verify delivery of their messages. Did the direct mail piece get read by the consumer? Was it even delivered? Did interested consumers view the television advertisement? How many consumers noticed the billboard? How many consumers read the newspaper or magazine notice? Market survey and research firms attempt to measure delivery, but their methods are necessarily statistical since they can't survey every household in the target area. Even with their limited usefulness and contested accuracy, these services are expensive.        The low success rates of direct mail (typically 2% to 4%) wastes much of the natural resources for printing and distributing the mailing, as well as wasting the money for preparation of the direct mail. Huge amounts of unsolicited mail end up in the wastebasket. Some authorities estimate that up to 70% of unsolicited mail is never opened.        It is expensive and cumbersome for providers to offer “considerations” to consumers. A consideration is a payment or award of some value to a consumer in return for a consumer viewing an advertisement or participating in a marketing survey. Sometimes considerations are given to encourage the consumer to an action, as when marketing research firms include a dollar bill in an unsolicited direct mail survey. Not only is there the cost of the consideration itself, there are the additional costs to the provider of identifying potential consumer recipients and preparing some means of delivery such as direct mail.        Providers have no practical way to get real-time (immediate) feedback on the success of their promotions. Marketing research on a particular product typically requires at least several weeks or months and is very costly. This leads marketers to test only large product groups and discourages them from gathering data about individual products.        Providers typically have no way to collect information about why a consumer purchased an item. Providers run special promotions, and consumers buy things, but it is arduous, tedious, and error-prone to draw the connection. It is difficult to judge the effects of promotions targeted to different sets of consumers, because providers don't know which promotion persuaded the consumer to buy.        Providers have few ways to collect information about “lost sales”, or why a consumer did not purchase an item. If lost sale data could be determined, a provider could better tailor offers to the consumer's needs and desires.        Providers attempt to measure and predict “consumer demand” to help determine the number and mix of products and services to offer and the prices to charge. Consumer demand is a measure of the number of consumers who want to purchase a product or service. It is typically calculated on a large scale with statistical models using historical purchase data. Demand can only be calculated based on purchases that have already occurred, since providers have few mechanisms to determine what new items consumers might want, or at what price consumers would buy. Demand information would be much more useful if it could guide providers into new territory, or if it could warn providers that a planned product would likely have few buyers or is being offered at an unsuitable price. With more accurate demand information, providers could plan inventories that are better matched to consumer desires, resulting in fewer markdowns.        Providers have no mechanism for using actual demand data to simulate consumer demand under varying conditions. It would be useful if providers could run “what if” scenarios to see the effects on demand of different prices, varying packaging, special discounts, etc. Using actual historical and current data to calculate this “theoretical demand” would be more accurate than using estimated or aggregate data.The Promise of Electronic Commerce        
The unexpected and explosive growth and popularity of the Internet in recent years has opened a new avenue for commerce—“electronic commerce”. Electronic mail (“e-mail”), the delivery of messages via electronic communication networks, has become a major notification mechanism, especially for point-to-point communications. Numerous “bulletin board” systems and the USENET newsgroup distribution network are popular broadcast notification systems. But it is the advent of the World Wide Web, frequently referred to as “the Web”, that has excited the imagination of thousands of consumers, providers, and entrepreneurs. The Web conveniently delivers text, images, and audio clips to individual users. In fact, the Web can be used to distribute any sort of information that can be represented in a computer data file.
The popularity of the Web has encouraged the establishment of many Internet Access Providers (LAPs), who provide communication access to the Internet for individuals and organizations; and Internet Service Providers (ISPs), who provide various services via the Internet, such as e-mail delivery, Web site hosting, search engines, and “chat” areas. This communications and information infrastructure continues to grow at a prodigious rate. With so many individuals and organizations now having convenient and inexpensive communications access, the Internet offers a promising base for a new mode of commerce.
Electronic commerce addresses many of the disadvantages of traditional commerce. It is convenient and inexpensive to prepare and deliver e-mail to specific persons or groups of persons. Many computer systems are repositories for immense databases that are useful for commerce, and the global communication network provides a means for accessing that data. Personal computer systems and specialized software are now enabling consumers to view online product catalogs and other information that providers publish on the Web. A multitude of researchers and organizations are working out the details of payment mechanisms to allow secure monetary transactions across the Internet.
The Disadvantages of Today's Electronic Commerce
Even with the colossal potential of the Internet, there are still a number of problems to be solved to support the establishment of a viable virtual marketplace, especially regarding the collection and exchange of market information. The electronic form of commerce doesn't address all of the problems of traditional commerce, and it raises a number of new difficulties.
Here are some of the disadvantages, from the consumer's point of view, of the electronic form of commerce:                Most search engines (Web sites that implement a capabilty for searching a database of information) are generic. They use general words as search keys, whereas consumers would benefit from information that is categorized by brands, product names, product category, store names, etc. Even though search engines are much faster than physical store visits, the search process is still tedious and prone to error.        Using search engines for comparative shopping is very slow. Search engines return pointers to information sites, not the actual information. Search engines frequently return thousands of “hits”, or items that partially match the search request. Consumers must sift through these hits, determine which ones are likely to be truly of interest, and contact the individual Web sites to collect the product information.        Consumers find comparative shopping tedious because every Web site has its own format for information. It is difficult to automate comparative shopping because of the inconsistent and non-standardized data formats.        It is difficult for consumers to find independent opinions about product quality, comparative features, and how a provider treats other consumers.        Every search starts from scratch. Even though some search engines now have the capability to narrow a search during an episode of use, each episode of searching starts anew requiring the searcher to enter all of the relevant keywords again. The preferences of the consumer are not retained between uses of the search engine.        Consumer searching is not private. The search engine can collect data about who is searching and the keywords of their search. Many Web sites maintain “cookies” or “passports”, that is, files that contain information about the consumer who is searching. In effect, the consumer's searching and decision making is exposed to public view.        Dealing directly with the provider exposes the consumer's identify and other data to the provider. When the consumer orders or purchases a product from a provider's Web site, the consumer must reveal name, delivery address, credit card data, etc. Even if the consumer is merely inspecting the information available on a Web site, the site's owner can still collect data about the consumer from the consumer's browser software.        Non-technical consumers may experience frustration in trying to construct appropriate queries for search engines. The syntax rules for queries commonly use Boolean logic and special separator characters. Even when the use of one search engine is mastered, the consumer must learn yet another set of rules for constructing queries for another engine, since the various engines use different Syntax rules for their queries.        Once a search engine successfully delivers a promising Web address (known as a “URL” for Uniform Resource Locator), the consumer may be disappointed to find that the URL is no longer valid. Thousands of Web pages are published and withdrawn daily, and the search engines are not always informed of the changes. Web pages customarily contain references to other Web pages (“links”), and a link is not always updated when the target URL changes, especially if the target URL names a page that is published by another entity. These “broken URLs” refer to Web pages that no longer exist or have moved to another address, so the consumer can no longer reach the information.        The style of presentation is still controlled by the provider. The consumer has no useful mechanism to request, for example, only summary information about products. The avenue of presentation is also fixed. If the data is available on the Web, it must be accessed via the Web; the consumer typically cannot arrange to have the data delivered, for example, via pager, facsimile, or cell phone display.        Usually there is no direct benefit to a consumer (a “consideration”) for viewing an electronic advertisement, although some companies have proposed paying consumers for reading advertisements via e-mail or other electronic delivery.        Consumers have few means for publishing their own electronic advertisements for buying or selling. Most newsgroups do not accept advertisements. Even in the special newsgroups that do accept advertisements, it wastes bandwidth since most readers of the newsgroups-won't be interested in a specific advertisement.        
Electronic commerce also presents a number of difficulties for providers, especially in the areas of deploying advertising and gathering “market intelligence.” Some of these disadvantages are:                Many providers are reluctant to advertise on the Internet because of software agents that make recommendations based on price alone. For example, the BargainFinder service, a research project of Andersen Consulting, that gathers pricing data on audio compact disks has been blocked from many providers' Web sites. Without the opportunity to present other features that justify a higher price, higher-priced providers would lose sales or be forced to lower their prices (and erode their retail margins).        Consumers may not be aware of a provider's Web site. A provider typically advertises the existence of a Web site via traditional means, driving up the cost of maintaining a Web presence.        Very few tools have emerged to help providers judge the effectiveness of their Web sites. For example, providers cannot gather enough information to calculate market share, since the statistics associated with competitor's Web sites are not publicly available.        Providers have little control over the search engines that consumers use to locate products, beyond carefully choosing a few keywords. They can't choose to emphasize different aspects of their products depending on the origin of the request.        It is impossible to collect useful demographic data about many online consumers, because consumers frequently use pseudonyms to disguise identity.        Due the lack of useful demographic data about online consumers, there is little guidance in targeting advertisements to potential customers.        Even though it is easier to personalize e-mail, how does the provider determine the target audience? Many online users summarily reject unsolicited e-mail advertising, disparagingly called “spam”.        Many Web sites now have the capability to generate Web pages (“content”) on the fly, but it is difficult for the Web server to obtain enough data about the requester to personalize the content in a useful way.        “Banner” advertisements placed on popular Web sites have not been particularly successful. Many online users don't “click-through” the banner to the more extensive advertiser information, because the placement of such banners is not finely targeted. Some Web activity statistics indicate that only one and one-half to three and one-half percent of users click-through.        Providers lose the goodwill of potential customers when they place advertisements in regular newsgroups.        Providers still have few ways to gather real-time feedback on special promotions or specific products.        Perhaps most critically, providers still have no method to derive useful market intelligence from the vast activity online. Providers still can't determine why consumers accept or reject offers, they can't calculate consumer demand, and they can't simulate demand based on actual demand data.Other Electronic Commerce Systems        
Much of the research relating to electronic commerce has been directed towards designing and implementing secure online money transactions. For example, U.S. Pat. No. 5,557,518 entitled “Trusted Agents for Open Electronic Commerce” issued 17 Sep. 1996 describes a system wherein a customer and a merchant can exchange electronic merchandise and electronic money by using trusted electronic agents. Even when this crucial aspect of electronic commerce is satisfactorily resolved, there remains the challenge of establishing a mechanism for the presentation, gathering, and exchange of market information in a way that encourages and supports broad participation in the online market. Of the current mechanisms related to online market information, most tend to fall into two categories: search engines, and various kinds of software or electronic agents.
Various problems with search engines have already been mentioned. Each engine has a different syntax and operation, making them error-prone and tedious to use. The information that engines return may be out of date or just plain incorrect. Search engines don't generally categorize information in a format that is handy for consumers, since they cater to general information seekers. Searches cannot make use of personal information about users, since the search engine has access only to the specific query data entered by the user.
A software agent is a software entity that is capable of performing certain delegated electronic actions (including holding information) on behalf of a user or another agent. An IBM white paper, “The Role of Intelligent Agents in the Information Infrastructure” (Gilbert, et. al., IBM Corporation, undated; also published on the Web at URL http://www.networking.ibm.com/iag/iagptc2.html; also hyperlinked from http://www.raleigh.ibm.com/iag/iaghome.html) describes three dimensions along which intelligent agents may be measured: agency, intelligence, and mobility. “Agency is the degree of autonomy and authority vested in the agent . . . Intelligence is the degree of reasoning and learned behavior . . . Mobility is the degree to which agents themselves travel through the network . . . ” (IBM, italics author's). Software agents can be further classified along the dimension of mobility into three broad categories according to their location of execution and location of data reference: mobile agents, “wandering” agents, and local or static agents.
True mobile agents are software entities that can electronically move from one computer system to another. The software program of a mobile agent actually executes on the target computer system. Although some technology to support mobile agents is available (for example, Sun's Java and General Magic's Telescript), they have not been successful, partly because many computer “firewalls” block the entry of mobile agents for security reasons, and because the agents must be capable of operating on a number of specific kinds of computers.
A wandering agent is a software entity that resides within a single computer system and “visits” or communicates with other computer systems, frequently via the Internet. Wandering agents are being used successfully to map the Web, gathering the data that is used in the internal indexes of search engines. However, these agents are very slow in operation, especially when there are thousands of sites to visit, and some wandering agents may be blocked from accessing some sites (as the BargainFinder agent has been). As described in “Internet Agents: Spiders, Wanderers, Brokers, and ‘Bots” (Cheong, Fah-Chun, New Riders Publishing, 1996), wandering agents are also used for various Web maintenance tasks and for Web mirroring. Cheong lists and describes many instances of wandering agents. The following list of wandering agents was compiled on 26 Dec. 1995 from Appendix G of Cheong. The purpose of each agent is excerpted by the inventor from short descriptions in Cheong.
NamePurposeASpider (Associativesearches for keywordsSpider)Arachnophiliacollect documentsAretha(none given)CS-HKUST WWWResource Discovery, validate HTMLIndex ServerChURLURL checkingCheckbot(none given)EIT Link Verifier Robotverify linksEmacs W3 SearchResource DiscoveryEngineFish SearchResource DiscoveryGetURLvalidate links, mirroringHTML Analyzercheck validity of Web serversHTMLgobblemirroringHarvestResourceInfoSeek Robotcollect information for databaseJumpStation RobotResource DiscoveryKatipolook for changed documentsLycosinformation retrieval and discoveryMOMspidermaintenance of distributed hypertextMac WWWWormkeyword searchingNHSE Web ForagerResource DiscoveryNIKOSResource DiscoveryNorthStar Robottextual analysis, indexingOpen Text Corporation(none given)RobotPeregrinatorindexingPython Robot(none given)RBSE SpiderResource DiscoverySG-ScoutResource DiscoveryScooterResource DiscoverySpry Wizard RobotResource DiscoveryTITALResource DiscoveryTarspidermirroringTcl W3 Robotvalidate linksTkWWW Robotfind logically related pagesW4 (World Widemeasure growth in WebWeb Wanderer)WM32 RobotResource Discovery, validate linksWWWW - World WideindexingWeb WormWebCopymirroringWebCrawlerResource DiscoveryWebLinkertraverses Web converting URN->URLWebWatchvalidate HTMLWebfoot Robot(none given)Weblayersvalidate, cache, maintain linksWebsnarfmirroringWebwalkResource Discovery, validate links, mirroring
A local or static agent is a software entity that operates within a single logical computer system, accessing data local to that system. Clearly this kind of agent is of limited usefulness for the electronic marketplace, since, a local agent would not have access to the variety of data that is necessary for a thriving marketplace. However, if there were a mechanism to incorporate data from many sources and to provide access to a broad base of users, local agents could be extremely useful.
These technologies have not been used to collect market information that providers can use to quantify consumer demand or to help gain customers at reduced cost. Consumers are hesitant to use some of these technologies because of privacy concerns. There isn't a practical mechanism for the user to instruct an agent to “keep looking” if the immediate search fails or is only partially successful. Although these technologies may be useful for the electronic marketplace, additional mechanisms are required for practical, ubiquitous electronic commerce.
The Fundamental Problems to be Solved to Enable Electronic Commerce
An electronic marketplace, just like a traditional marketplace, must support the basic process of commerce: offers to sell or buy are made, offers are accepted, and considerations (payments) are paid. If the basic process does not work, there is no marketplace. A viable marketplace must also address side-effects of commerce such as issues of security, privacy, and confidence or trust; otherwise, even if the basic process works, consumers and providers will not feel comfortable enough to participate in the marketplace.
A practical and viable electronic marketplace involves the exchange of market information, as well as the more obvious trading for goods and services. From a consumer's point of view, shopping is a means of gathering data about goods and services offered. This data is used by the consumer to compare and rank offerings and to make decisions about purchases. From a provider's point of view, consumer shopping is an opportunity to gather data about consumer needs and interests. This data is used by the provider to improve product and service offerings.
For consumer, the fundamental problems relating to the flow of market information in electronic commerce, still to be solved, are:                Consumers need help in gathering information about available goods and services, quickly and with a minimum of fuss;        Consumer privacy needs to be assured while gathering information about available goods and services; and        Consumers need a mechanism for ongoing, autonomous searches for information about available goods and services, searches that continue even when the consumer is not “on-line”.        
For providers, the fundamental problems relating to the flow of market information in electronic commerce, still to be solved, are:                Providers need to target advertising information to truly interested consumers without disturbing the privacy of those consumers;        Providers need to be able to quantify consumer demand, both offline and in real-time, using historical and current data;        Providers need to be able to determine reasons for sales and lost sales; and        Providers need a source of more accurate market data to serve as input to present and yet-to-be-developed market analysis methods.        Solutions to these problems for providers must be cost-effective.        